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ABSTRACT 

The study examined the quality cf educational 
research published in the journals, with a focus on the following: 
(1) an evaluation of the quality of contemporary published research 
from the standpoint of sound research conduct and reporting as judged 
by experts, and (2) an examination of the relationship between 
assigned quality ratings and selected characteristics of research 
articles and participation exports. A stratified random sample of the 
1971 educational research articles was selected and a sample of 
judges to rate the articles was selected via the membership directory 
of American Educational Research Association (AERA), The results of 
the study provide consumers of research with needed information 
regarding the soundness of the research whose findings influence 
present-day decision making. (Author) 
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Introduction 

This study uas designed to determine the quality of educ^:ticnal rce^firch 
articles published in educational journals and in Journals of related pro- 
fessions. Published educational research is the delivery system by T/hich pro- 
fessional educators acquire much of their understanding of today's educational 
problems. In this critical cominunicative process, the need for sound research 
yielding;, meaiiinpf ul, interpretable inforiration is inperative. Yetj consider- 
able doubt has been raised as to the quality of published educational research 
artd^ therefore 5, as to the validity and significance of the data being provided 
educators via the research reported (Eloon, 1966; i'ichael, 1963: Scrlven, 1960) 
The only recent comprehensive study of the quality of published educational 
research \ as carried out on a representative sarple of 1^62 research articles 
by an ad hoc Corjmittee cn Evaluation of Educational Research established by 
AEI<A (Ifandt* 1SG7) • Tliat study found that a majority of research articles 
published in 1S62 contained serious flav;s* jlore disturbing, less than 7% 
of the research articles published in education journals were rated as being 
i^ortb.y of publication, and the qual:'ty of research published in education 
journals ^;as found to be narkedly inferior to that published in journals of 
related professions - primarily psycholop,y. 



^Presented at the American Educational !ver>enrch Association Annual Conference 
Chicago, April 1974. 

^University of South Florida 
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numerous guidelines for evaluating educational research have appeared 
in the last 20 years, attesting to the need for informed, critical evalua- 
tion of the material appearing as research (e.f^. Best, 1970; Borg, 1963^ 
Dvorak, 1956; ^arquhar & Kruraboltz, 1959: Johnson, 1957' Kohr & Suydam, 1970: 
Strauss, 1969 Symond, 1956: Van Dalen, 1950: I.^iersma, 1969). Hov/ever, empiri- 
cal studies on the question of the quality of actual research have been almost 

non-existent. Ko follow-up of the 1962 work of the AERA Committee on Evaluation 

f 

of Research has heretofore been undertaken. 

The years since l'^62 have brought many ostensible advances in educational 
research. Membership in AERi\ has grovm tremendously since 1962; nev7 journals 
have been established to publish educational research. Educational research 
is no^7 recogni?;ed as a specialty in its own right, not simply a subspecialty 
of psychology. The years since 1962 have also been the years of federally 
funded projects, in educational research and for programs with a mandate for 
"evaluation." A consideration of these changes led to the decision to re- 
plicate the earlier study. 

Statement of the Problem 
The purpose of the present study ^ras to determine the quality of the 
current body of educational research published in journals. In examining 
the genaral question of quality of published educational research, tho investi- 
gators focused on the following specific questions: 

1. Vfnat per cent of the educational research articles published in 
journals are considered by experts in educational research to 
(1) merit publication without change, (2) need minor revision to 
make them acceptable for publication, (3) need major revision to 
make them acceptable for publication, (4) be so low in quality they 
should not have been published? 

2, How do the experts rate representative educational research articles 
on specific characceristics related to the quality of research and 
of research reporting? 
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3. Is there a difference in quality, both overall and on the specific 
characteristics, between educational research articles published in 
(1) research - oriented education journals, (2) non-research oriented 
education journals, and (3) related profession journals? 

4. l^Jliat specific shortcomings are most frequently cited by experts to 
substantiate a judgment to reject or require major revisions in an 
article. 

In order to compare the findings with those of a decade ago, the study 
was, insofar as feasible, a replication of the x^ork of the Committee on Evalua- 
tion of Research (Wandt, 1967). Procedural departures from the earlier work 
v7( re taken v;herc necessary in an attempt to strengthen the research design. 
Sc r.e changes were also required because of changes over the decade in the 
potmlation of journals, and reviewers. Ilany of the procedural changes were 
based on recommendations of the earlier investigators (VJandt, 1967) or insp.'ired 
by difficulties that they encountered. 

In addition to pursuing the basic questions posed above, the investiga- 
tors also examined the influence of selected characteristics of the judges 
on assigned ratings. 

Methodology 

The sampling and data gathering procedures of the study were identical 
to those of the earlier study of the committee on Evaluation of Research except 
where specifically noted. 
Selection o f journals a nd articl es. 

The first task v;as the identification of the population of educational 
research articles and selection of a representative sample of the articles. 

In the original study the following criteria ^^rere established for iden- 
tifying journals publishing educational research: (1) They must have been 
indp.xed in Education Index and (2) They must have been cited 10 or more times 
in chapter bibliographies of the Review of Educational Research during the 
gj^Q full three year cycle immediately preceding the year selected for study. 
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RER changed its edicorial policy in the middle of 1970, so that the last 
full 3-year cycle v/as from mid-1967 to mid-1970. In the future, this may not 
be a useful criterion to use in a study of this kind. Perhaps citation in 
Encyclopedia of Educational Res < ^arch or the Arnual Aeviet; of Educational 
Research could be used. 

Journals were classified as "Related Profession" (RP) if they were in- 
dexed in Ulrich's Periodicals Diractory under the headings of psychology, 
sociology, and medical science. Tht: remaining journals were classified as 
either "Educational, publishing primarily research'' (ER) or "Educational, 
publishing primarily non-research" (NR) depending upon the per cent of 
articles (exclusive of reviews and comments) that were research*^ The 
criterion for classification as ''primarily research" was 51% or more of 
the articles devoted to research. The 1971 population of journals is re- 
ported in Table 1 and the population of journals for the 1962 study ±g in 
Appendix D. 

In comparing the population of journals with that of the earlier study, 
the folloxv'ing changes were found; 

(1) Fif ty-se'^'^en journals met the selection criteria in 1962, only 
4o met tb.em for 1971. Nine journals in 1962 and two in 1971 
had no research articles, leaving a population of 49 for 1962 
and 44 for 1971. 

(2) In 1962, only 5 journals x/ere classified as "Educational, pub- 
lishing primarily research," In 1971 there were 13 - a dramatic 
change. The number of WR journals decreased from 27 in 1962 to 
22. There was no change in the number of RP journals. 

(3) Several journals selected in 1971 were classified differently 
than in 1962: Three NR (1962) were ER in 1971: tv;o ER (1962) 
were MR (1971) and one RP in 1962 became ER in 1971. 

(4) Of journals added to the list, 6 were ER, 11 MR, and 4 RP; of 
those deleted, 1 vj?s ER, 15 NR, and 3 RP. 



*The 1962 study used "percentage of total journal pages devoted to res<ir 
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One unanticipated difficulty from this objective process of selecting 
articles was that five articles from three "Related Profession" journals 
(American Journal of cental Deficiency , Journal of Speech and Hearing Res earch, 
Journ a l of Speech and Hearing Disorders ) were so technical in a non-educa- 
tional specialty that the rGviex>?ers w<^ve unable to review them. For each 
such article, a replacement was randomly selected from the same journal. ^ 
In future studies of this type, perhaps a criterion of "relevance to educa- 
tion" should be added to the selection criteria for the population of articles. 

The sample was formed by stratifying the copulation of articles (N 1436) 
by journal, and drawing an 8% sample of articles at random within each stratum 
(journal), thereby creating a proportionate stratified random sample of 121 
^articles. (See Table 1). The 121 articles are listed in Appendix C, The 
earlier study sampled from a population ot 327 research articles. The 1486 
research articles in the 1971 population represents a marked increase in 
number from a decade earlier, although the 1971 population is contained with- 
in fewer journals (44 as compared to 49), 
Sele ction of Judges 

Judges for the 1962 study were selected from the AERA directory on the 
basis of their 'professional reputation in educational research mecnodology. " 
The investigators in the present study decided to begin with a random saiaple 
of the neinbers in Division D (lleasurement and Research Ilethodology ) of the 
1971-72 AFRA Directory, The first step was the selection of a 10% sample 
of the approximately 5000 regular members of Division D, selecting the first 
person randomly, then proceeding to select every 10th person in the directory. 
The approximately 500 Division D members selected were then sent reply cardr; 
inviting them to serve as judges. Of the 439 members who responded, 353 
indicated a willingness to participate and supplied information about their 
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The judges in the original study were described as being selected on the 
basis of their professional reputation. Ninety-eight per cent held the (Joe tor- 
ate, 63% had taught research courses, 75% had supervised a dissertation, and 
78% had supervised a thesis. The median number of publi£^hed articles was 16. 
The volunteers for the present study departed soraex^hat from the original group 
of judges as to these characteristics, so an attempt was made to select from 
t&e pool of volunteers a group which was as similar as possible to the orig- 
inal group of judges. After first being selected in regard to "earned doctor's 
degree*' the volunteers were rank ordered as to "number of publications." An 
attempt was then made to select those with the greatest number of publications. 
The final criterion was supervision of theses and dissertations. Even with 
this deliberate selection, the present sapiple of judges has fewer publications 
and a little less research experience than judges for the earlier study, perhaps 
indicating that they are generally a younger group than those in the 1962 study. 
The final group of Judges selected numbered l7l; 121 to serve for the main rating 
study and 50 to serve in a reliability study of the ratings. The characteris- 
tics of the judges are reported in Table 2, 

The great majority of judges in the earlier study (84%) were members of 
the American Psychological Association. A post hoc check of judges in the 
present study found only 44/' to be members of APA. The judges are listed in 
Appendix B. 

Assi^n me n t of A rtic les to Judges 

One of the 121 articles to be evaluated was assigned at random and mailed 
to each of 121 judges drawn randomly from the total group of 171 judges. 
Following a recommendation in the earlier study, the investigators attempted 
to reduce bias on the part of the judges toward the articles by having all 
articles reproduced with the name of the Journal and the names and addresses 
of the authors omitted. The remaining 50 judges were each randomly assigned 
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and mailed one of the 121 articles being evaluated by the larger group of 
Judges, T])e pairs of ratings thereby derived on 50 of the 121 articles x^as 
the basis for a reliability study of the ey^luative ratings, A check on the 
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assignment process revealed that- in no ca^e did a Judge receive an article 
of which he was an author. 

Ar. the time of this report, rating data had not been received on seven 
of the 121 articles. Any resulting bias J.s assumed to be minimal because the 
missing ratings are distributed across six different Journals and all three 
journal categories. However , for the data analysis reported below, the miss- 
ing ratings reduced the U_ of the basic study to 114 and the reliability 
to A4. 

Th e Evaluation Instrument 

The evaluation instrument used was a modification of that devised by 
the Committee on Evaluation of Research (Wandc, 19o7). It required three . 
types of reactions to the article: (1) ratings on specific characteristics, 
(2) overall ratinp, and (3) justification of overall-' rating. 

Each judge was asked to rate his assigned article in terms of ^3 charac- 
teristics deemed desirable as aspects of quality in conducting and reporting 
researcli. For each characteristic, a five-point scale wa.. used, representing 
live levels of quality: 

Level of Quality De script:ion 

5 - Excellent A model of good practice 

4 Oood A few minor defects 

3 - 'ediocre Not good, not bad 

2 - Poor Soine serious defects 

1 - Complete] y^.j A horrible example 
Incompetent 

If a characteristic was not appropriate to the research study, the 
judge ^;as asked to place an ' X' by the characteristic. 
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The 33-item rating scale wasr an expansion of the 25-item scale used in 
the earlier study* The investip.ators added ^.eigh^ items to the original scale, 
two that were suggested by the earlier study and^^ix that arose from the in- 
vestigators' personal experienc;^ in research consumption. The 33-item scale 
\>jas composed of seven subscales covering the following topics: (,1) Title 
(2) Problem, (3) Review of Literature, (4) Procedures, (5) Data Ar\aly3is, 
(6) Summary. and Conclusions, and (7) Form and Style. The rating scale is 
included as Appendix A. 

Each judge was next asked to assume the role of editor of a journal that 
published educational research and to makfe one of four choices in regard 
to his assigned article: (1) accept as i^ for publication, (2) accept for 
publication after minor revisions, (3) accept only after major revisions, 
or (4) reject it. This 4-choice rating is hereafter referred to as the ARRR 
rating. Those judges who rated their assigned article as (3) or (4) were' 
asked to indicate which of the 33 specific shortcomings they would cite to 
substantiate their judgment. 

In the original study the judge had only three choices in his role as 
editor: (1) accp.pt as is, (2) accept after minor revisions, or (3) reject. 
The earlier investigators found that many judges stretched category (2) to 
Include major as V7ell as minor revisions. Therefore, ^'accept after major 
revision'" was added as an explicit category in the present study. 

Interjudge Reliability of the Instrument 

For 44 articles ratings were received from two judges. Inter-judge corre- 
lations were computed for all subscales, for the total scale, and for the ARRR 
ratings. These correlations, reported in Table 3, range from ,11 for the 1- 
item scale of ''Title" to .GC for the 5-item scale of "Review. of the Literature". 
In addition, a check was made as to the extent of agreement between pairs of 



judges on the ARRR scale. As Table 4 indicates, there were no cases in which 
an article rated "Reject" by one juage was rated "Accept as Is" by the other 
and vice versa. There were only 9 cases (20%) in which the judges diffq^d 
by twtj categories. Thirty per cent of the judgments were the same and an 
additional 50 per cent differed by only one category. 

Internal Validaticn of the Instrument ^ ^ 

Since there is no external critv?rion against which to validate the 
rating scale, an attempt was made to gauge the consistency of the ratings 
within the instrument. A total scale score was computed for each article, by ^ 
adding the numerical values of the 33 ratings and .dividing by the number of 
items reted.2 Similarly, a mean rating was computed for each of the seven 
subscales of the rating scale- The internal, consistency of the instrument 
was checked in two V7ays : ' " - \ 

1. Computation of intercorrelations of all the subscales.,; Except for 
correlations with "Title , these ranged from .30 to .65. All inter- 
correlations were significant (df = 112) beyond the .005 alptja level. 

2. Computation of the tetrachoric correlation between the total 
scale score and the ARRR rating, on which raters judged the 
article on a 4-point scale. The ARRR ratings were split so that 
categories 1 and 2 formed one gtoup and categories 3 and 4 formed 
the othet group. The r was .87 (df = 112) which is signifidant 
beyond the .005 alpha level. 

f'rora these analysoe it was concluded that the rating, scale -t^as sufficiently 

consistent Mthin itself to be useful in rating research articles. 

Findinga-' 

Before the data vrere analyzed, a check was made to see whether the 
characteristics of the judges were related to their ratings. It vras found • 



2 The total scale score for some articles was based on souewhat less than 33 
items because one or more items were marked "hot applicable by the judge. 

^ 3 Findings of the 1962 study (Wandt, 1937) are reproduced in Appendix D- 
cnir^" Appreciation is expressed to Edwin Wandt for granting permission to repro- 
LiMmi^miM duce these findings. \ 
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that there was a tendency for non-APA members and less experienced judges (i.e. 
no research teaching, no dissertation supervision, no thesis supervision) t:) 
rate articles slightly higher than the more experienced judges did. However, 
it was also found that the reviewers with varying characteristics were, quite 
well distributed among the three types of journals, so it was decided to Ignore 
characteristics of judges in further analyses. 

Overall Quality 

The findings on the ARRR ratings for the 114 educational tresearch articles 
for all journals, fbv all education journals, and across the three journal 

0 

categories are suramarized in Table 6. 

For all journals, only 8% of rr ^arch articles were rated '^Acceptable as 
is for publication", ^^ 31% were rated ^'Acceptable after iPinor revisions", 34% 
x7Gr^ rated "Acceptable only after major revisions", and 27% were rated ''Reject*'. 

A chi-square analysts was made on the distribution of ARRR ratings across 
the three journal categc/ries,^ ERj NR, and RP. A chi-square va^ue of 2.50 was 
obtained (df = 6), which was non-significant. The distribution of ratings 
on overall quality was, therefore, interpreted to be comparable, regardless 
of the type of journal. 
Ratings cn t he 33 Characteristics 

The findings on 33 specific characteristics of the 114 educational re- 
search articles for all journals, for all education journals, and across the 
three journal categories are summarized in Table 7. 

Tor ?11 journals the highest mean rating was assigned to ''Title is well 
related to content" (3.80) and. the lowest was assigned to "Limitations of the 
study. are stated" (2.37) followed closely by "Validity and reliability of data 
gathering procedures are established (2.43), No mean rating fell belov; 2.00 
in any journal category and only one mean rating was ac 4.00 or above. The 
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midian rating v/as 3.26. * 

For all Education Journals the mean ratings ranged from 2.21 to 3,55, 
with n median of 3.00, 

On the following; characteristics., the assigned ratings were ^lonsiderahly 
belo\7 the ^median for all three journal categories: (1) Limitations of the study 
ar,e stated, (2) Validity and reliability of data gathering procedures are 
established; (3) Studies are examined critically, (4) Assumptions are clearly 
^statedj (5) Research design is free of specific weaknesses, and (6) Method 
of sampling is appropriate. 

Th^t^ollowing characteristics received ratings considerably above the 
median for allgithree' journal categories: (1) Title is well related to article 
(2) Problem is significant, (3) Source of important findings are noted, 
(4) l}ata gath<Bring methods are described, (5) Conclusions are relevant to 
the problem, (6) Report is logically organized, and (7) Tone of the report 
displays an unbiased attitude. The mean ratings of the 33 characteristics 
for the articles in ER journals ranged from 2.19 to 3.60, with a median of 
3.04. For the articles in NR journals, the mean ratings ranged from 2.16 to 
3.76, with a median of 2.90. For the articles in the RP journals, the mean 
ratings ranged from 2.54 to 4.13, with a median of 3.45. 

Ra'inp.s .Qn_the Subsc ales 

The mean and standard deviations across the rating subscales are pre- 
sented for the three journal categories and for all journals in Table 8a 
and the results of a Type 1 ANOVA for Subscales X Journal Categories are 
summarized in Table 8b. 

The AKOVA V7as p rformed using Journal Category as a between-subj ects 
variable composed of three levels (ER, NR, RP) and using Subscales as a 
within-subjects variable composed of seven levels (Title, Problem Statcjr.ent, 
Review of Literature, Procedures, Data Analysis, Suimnary and Conclusions, and. 
Form and Style). Ihe ANOVA revealed a main effect on Journal Category 
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significant beyond the .001 alpha level and a main effect on subscales also 
significant beyond the .001 alpha level. There was no significant interaction 
betTveen the two variables. 

The post hoc analysis for the subscale means indicated that the differ- 
ence between A (Title) and G (Form and style) is not significant, while 
the means for both of these scales differ from those for the other five 
subscales. Tlie means of the other five subscales do not differ significantly 
from each other. For the journal categories, the post hoc analysis indicated 
that the mean for "Related Profession" journals is significantly higher than 
that for both categories of Education journals, but the difference between 
the means of the tV70 categories of Education journals is not significant. 

Specific Shortcomings C'.ted by Judges 

Each judge who, in the role of edi;:or, chose to reject his article 
or to accept only after major revisions ras asked to indicate tha specific 
shortcomings he would cite to substantiate his judgment. A tabulation of 
the resulting citations is presented in Table 9 for all journal articles 
and for articles in each of the three journal categories. For each charac- 
teristic Table 9 contains the frequency of citation and the rank within 
each journal category. 

The 10 most frequently cited shortcomings of articles in All Journals 
in order of frequency and with per cent of articles affected x>rere: 
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1. Research design is free of specific weaknesses (27%), 

2. Research design is appropriate to solution of the problem (23%). 

3. Validity and reliability of data gathering procedures are estab- 
lished (22%). 

<4. Conclusions are substantiated by the evidence presented (22%). 

5. Hethods of sampling are appropriate (21%). 

6. Appropriate methods are selected to analyze data (21%). 

7. Conclusions are significant (20%)* 

8. Limitations of the study are stated (19%). 

9. Tables and figures are used correctly (18%) • 

10. ?,esults of the analysis are presented clearly (16%) • 



The pattern of shortcomings in the total sample of articles was generally 
ccn:ii:cn across all tliree journal categories. A Kendall coefficient of concor- 
dance (W) computed on the three sets of ranking was found to be .71, indi- 
cating a significant (pC.OOl) relationship among the rankings across the 
three journal categories. Furthermore, seven of the 10 most frequently 
cited shortcomings in the All Journals category v/ere also among the 10 
most frequently cited in each of the three separate categories. However, 
disparities were apparent across the seven subscales of the rating instru- 
ment. Virtually all of th3 most frequently cited characteristics were 
contained x/ithin the following three subscales: (1) Procedures, (2) Data 
Analysis, and (3) Summary and Conclusions. Those characteristics rele- 
vant to Title, Problem, Review of Literature, and Form and Style v/ere 
relatively free of frequent citations. 



Linitations of the Study 

The findinps and conclusions of this study should be interpreted in light of 
the foiloTTlnf: linitations: 

1. As in the ori<^inal study, the articles were evaluated by judges who vrere 
.Considered expert in educational research but not necessarily in the 
specific area of the study v^hich they evaluated. Matching specialty of 
the article to specialty of the judfre might have produced somewhat dif- 
ferent results. 

2. The population of articles in the study was created usinf: an arbitr-^ry, 
though objective, operational definition of "educational research*^ Un- 
doubtedly, a chanf;e in this definition v;ould alter the population make- 
up and possible result in a population \;ith characteristics different 
from the one in this study. 

3. Generalizations from this study .should be limited to articles published 
in t'le vear 1971. Although it is reasonable to suggest that similar 
patterns may exist for articles published during other years, no evidence 
bearinr, on this point is at hand. 

A. Although "blind" ratings T^ere used, some judges may have recognized 

the source of the article, so that some degree of bias may have existed 
vhich is undetected. 

Discussion and Conclusions 
This survey of the quality of published educational research v^as a replication 
of a studv conducted in 1962 by an , ad hoc committee of AERA. Interpretation of 
the findings ne^cesj^ar ily requires some comparisions with those of the earlier study. 
These comparisons must consider three variables: (1) The "true" quality of the 
articles, (2) The biases of the judges for or against a class of journals, and 
(3) The reference point or rating standard of tl^fe^udges. 
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The real concern of the lnvGsti<>ator s was \;ith the first variable, that is, 
the **true" quality of the research. IIovTever, all conclusions reparding the quality 
of the research nuf^t be tentative because the effects of the other tuo forces are 
unkno^/n at the present tine* 

If the possible problems of judp,e bias and judf^e standards is Ip.nored, the 
folloT7inp, conclusions rmav be dravTi: 

1. The percentage of published articles rated "reject" was lov^er for 1971 
(27%) than for 1962 (40%), but the percentat^e rated "accept as is" is 
also louer (9% in 1971, 19% in 1962). These data are confounded by the 
chanpe in the rating scale fron a 3-point scale in 1962 to a 4-point 
scale in 1971. If the "llinor revisons" category for the 1971 study is 
collapsed i;ith "Accept as is", then the percentage of acceptable articles 
is still only 39%. 

2. The superiority of the 'Related Profession' journals over "Education" 
journals is not as apparent in this study as it x;as in the 1962 study. 
There v^as no difference in the percentages of the four ARRR ratings 
assigned to each category of journals in this study. For the tncv^ns of 
the ratinps on the 33 specific characteristics, the difference betx^^een 
educatioii and RP journals, although significant, is less pronounced than 
in 1962. The median of the item ratings for the tvo categories of 
education journals are very close to those for 1962, v?hile that lor RP 
journals is lo-rer than for 1962 (3.45 for 1971, 3.92 for 1962). 

3. The ouality of published educational research is still mediocre. That is, 
the medians of the item ratings for all journal categories x;ere approxi- 
mately 3.0, defined in this study as "Mediocre, not good, not bad". 

4. The rreatest deficiencies of articles xrere in characteristics related to 
"procedures", "data analysis", and "suirmary and conclusions". The 
specific characteristics with the loxTcst ratings generally cane fro^a 



these sections, as did tl\e tnont freouently cited shortconlngs . Althoup,h 
an open-ended procedure \;as used to identify shor tconinf.s in the 1952 
study vhile the present study used a structured response, it may be of 
interest to conpare the ten specific deficiencies cited most frequently 
in both studies. Two of the deficiencies cited most frequently for 1971 
articles vere not included on the 1962 scale and \;ere mentioned rather 
infrequently as deficiencies in that study. Five deficiencies were among 
the top 10 for both studies: 

a. Validity and reliability of data cratherinr, procedures are established 

b. Conclusions are substantiated by the evidence presented 

c. Methods of samplinp are appropriate 

d. Appropriate methods are used to analyze data 

e. Results of the analysis are presented clearl'^. 

All of these characteristics are critically important to consumers of 
research to assure proper interpretation of research findings. 
The precoflin^ conclusions are based on two assumptions. The fdrst is that 
the judo;es in the present study used essentially the sane standards of excellence 
as '^ere used by the judges in 1962. This assumption is supported by the .similarity 
of judj^e characteristics and the use of virtually identical procedures in defining 
and sampling articles and in fathering the rating data. The other assumption irs 
that no biases for or ^igainst the v^rious journal categories existed in either 
study. Since an attempt was made to remove this source of bias in the present 
study T'hile it is an unknovm for the earlier study, this assumption is more 
difficult to support. Since neither of these assumptions v;as tested, several 
alternative conclusions may be draTm. 

1. Research published in RP journals nay have declined in quality, T;hile 

that published in educational journals has remained constant in quality, 
Q '^his vould tie the case if neither -judge bias or changing standards were 
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operatinr in either stud' . 

2. Research published in both educatioi.-^l journals and PvP journals has re- 
mained constant in quality. This v;ould ^ ^ the case if judge standards 
remained constant and a selective bias favorinrr RP journals in the 1962 
5^,tudy T'as eliminated in the present study by the use of blind ratings. 

3. The quality of research in RP journals may have remained constant and 
the quality for educational journals may have declined. This i/ould be 
the case jf there were bias on the judpes' part in the earlier study in 
favor of the PnP journals and ap.ainst the educational journals, and if 
the ratinps standards remained constant. If the blind ratinp,s in the 
present study eliminated these biases, then the ratinf^s obtained in the 
present study rould anpear to support this conclusion. 

A. All published research may have improved in quality since 1962. This 
\70uld be the case if judges' standards have been raised and if selec- 
tive bias favorinp RP journals but not nep.ative tov/ard educational 
journals operated in the 1962 study but not in this one. 
In the absence of more information it is impossible to decide which is the 
most acceptable conclunion. Additional studies are required to resolve this 
nuestion. 
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Table 1 

Population and Sample of Ed-jcitional T.esearch Articles Published in 19 71 



vTournal 



No. of Articles 



In Ponulafion 



hd 


ucation Journals with Primarily 


^.esearch Articles 




559 




1 . 


AV Communications Review 




15 




1 


I . 


American Educational Research 


Journal 


35 




3 


3. 


British Journal of Educational 


Psychology 


37 




3 


4. 


Counselor Education and Supervision 


23 




rs 


^ • 


Educational and Psychological 


' leasurement 


40 




3 


6. 


Exceptional Children 




58 




5 


J. 


Journal of College Student Personnel 


5? 




5 


8. 


Journal of Educational Ileasurement 


30 




2 


9. 


Journal of Educational Research 


81 




6 


10. 


Journal of Experimental Education 


56 




5 


11. 


Journal of Negro Education 




27 




2 


12. 


Journal of Research in Science 


Teaching 


io 




4 


13. 


Psychology in the Schools 




A? 




4 


Education Jourrals with Priniarilv 


Won- 2 ea r cji_^ Ar t i 








1. 


T^American Vocational Journal 
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2. 
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Comparative Educational ^^evle^^ 
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i 


5. 


Educational Leadership 
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1 


6. 


Elementary School Journal 




11 




1 


7. 
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O . 
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*^ 


10, 
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11, 
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if- lathematics Teacher 




6 




1 


13. 


Personnel and Guidance Journal 
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18. 


Science Education 
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2 
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22. 


T^rVolta r.eview 
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1 



In SaiT^nle 
T5~ 



21 



P.elated Professions 

1. American Journal of Mental Deficiency 

2. American Journal of Orthopsychiatry 

3. Child Developraent 

4. Journal of Applied P:;ychology 

5. Journal of Counseling Psychology 

6. Journal of Educational Psychology 

7. Journal of Speech and Hearing Disorder 
3. Journal of Speech and Hearing Pesearc!^ 

Sociolocyy of Education 



6^)7 



All Journals 



109 


5 


19 


2 


12C 


10 


94 


8 


97 


8 


77 


6 


3S 


3 


96 


8 


17 


1 



I o ^ 



121 



* Ten journals published less than 7 research articles each in 1971, The 30 articles 
f*om hcse journals were treated as a single group for purposes of sampling. A total 
O j 3 articles were chosen from this group with the restriction that no more than one 
ERJC tide could come from any given journal* As a result, 7 journals x^ere excluded 

iiriimimffi'ii|iio,Qjjj SamplC* 



Tabic 2 

Characteristics of the Sample of ^^f kV.RA Division D Members 



Percent 



10 



90 



Has earned doctor's degree 

Has taught course in r:ethods 
of educational research 

Has supervised doctoral dissertation 

Has supervised master's t!\esis 

Has served as a reviev; edicor of 
a journal 

Member of APA 

Years of ' full time equivalent:' 
research experience 

Mumber of educational research 
articles published 



100% 
73% 

co% 

56% 



COO 



1G.03 



2':h3 10.00 



3.81 AC. 10 
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TABLi: 3 



luteriudge ''.oliability Esti^iatcs for Subscale Scores, 
Total Scale Score, and ARrj< Score for 44 Articles 



Variabla r Si^>nif icance Level 



Subscale Scores 



Title 


.11 


NS 


Problert statenent 


• 18 


ilS 


?.evie;7 of the literature 


.63 


,01- 


Procedures 


.37 


,05 


Data analysis 


• 26 


MS 


Suroniary and conclusion 


• 17 


MS 


Forn and style 


.12 


NS 


Tot'-^l Scalo Score) 


.43 


• 01 


ARRP. Score 


• 21 


US 



TABLE 4 



Agreement of First and Second Judges 
on ARRE Ilatings for 44 Articles 









Rating of Firct 
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Accept 


Accept aft3r 


Accept 


after 










as is 
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Hajor 


ReviGions 


Reject 
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Accept after 
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Accept after 
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Accept as is 
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0 
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Intercorrvilations of the 
Suh scales of the Rating; Scalo 



Subscales 



A. Title 

B. Problen Statement .29 

C. Literature Reviev .15 .40 

D. Procedures .28 .57 .37 

E. Data Analysis .25 .49 .33 .6^5 

F. Summary, ConrT usions ,22 .56 .30 .51 .56 

G. Form & Stvle .27 .51 .45 .65 .5o .60 
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Tabic: 6 

Accept - Revisa (r.iinor) - Ilevise Cnajor) - Pcject Ratings of llA 
Eaucational Research Articles Published in 1571-^ 



Kumber of Percent rated as 

articles in 

Source of the articles sample Accept iiinor re- rlajor re- 

as is visions visions Reject 







n 




n 




n. 


% 
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"/ 


All educatibnal journals 


63 
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(6) 
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17 
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Education journals v?hich 
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> 


(S) 
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(31) 


39 
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31 


(27) 


^ for distribution of ratings 


by 


journal cateo;ories 


= 2. 


50 1 di 


= 8; WS 







See Table 1 for journals m each category. 
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I. Evaluation of Specific Characteristics 
A. Title 

(1) Title is well related to content 

of article 
K • Probleni : 

(2) Prcblep is clearly stated 

(3) l:3'potheses are clearly stated 

(4) Problem ia si<^nificant 

(5) Assunptions are clearly stated 

(6) Limitations of the study are stated 

(7) Ir.iportant terras are defined 

C. Review of literature 

(8) Coverage of the literature is adequate 

(9) Review of the literature is well organized 

(10) Studies are exaniined critically 

(11) Source of important findings are noted 

(12) Relationship of the problem to previous 

research is made clear 
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D. Procedures 

(13) Research design is described fully (13) 

(14) Research design is appropriate to solution (14) ] 

of the problen' ^ ■ 

(15) Reserirch design ±s free of specific (15) - 

weaknesses 

(16) Population and sample are described (16) 

(17) I'lethod of samplinr; is appropriate (17) 
(15) Data gathering methods or procedures are (18) 

described | 

(19) Data gathering methods or procedures are (19) | 

appropriate to the solution of the problem [ 

(20) Data gathering methods or procedures are (2C) | 

used correctly | 

(21) Validity & reliability of data gathering (21) \ 

procedures are established 

^ • Data analysis 

(22) Appropriate methods are selected to analyze (22) j 

data ! 

(23) Methods utilised in analyzing the data are (23) 

applied correctly 

(24) Results of the analysis are presented (24) 

clearly 

(25) Tables and figures arc effectively used 
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(25) 

F. Summary & Conclusions 

(26) Conclusions are clearly stated (26) 

(27) Conclusions arc substantiated by the (27) 

evidence presented. 

(28) Conclusions are relevant to the probleni (28) 

(29) Conclusions are significant (29) 

(30) Generalizations are confined to the popu- (30) 

lation from which the sample was drawn 

G. Form 6, Style 

(31) Report is clearly written (31) 

(32) Report is logically organized (32); 
(31;) Tone of the report displays an urbiased, (33)i 

impartial, scientific attitude ' 



_ 5 4. 

^ i 



2 1 



-i.. 



-1 



5 


4 


3 


_2_ 1__ 
















1 






- - i 



II. Overall Evaluation (check one) 
(1) Accept as is 



_(2) Accept after minor 
revisions 



_(3) Accept after major revisions 




-(«4) Reject 



C. Rev^cv of literature 

(8) Coverage of the literature is adequate (B) 

(9) Review of the literature is well organized (9) 

(10) Studies are examined critically (10) 

(11) Source of important findings are noted (11) 

(12) Relationship of the problem to previous (12) 

research is made clear 
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Procedures 

(13) Research design is described fully (13) 

(14) Research desi^^n is appropriate to solution (14) 

of the problem 

(15) Research design is free of specific (13) 

weaknesses 

(16) Population and sample are described (16) 

(17) ilethod of sampling is appropriate (17) 

(18) Date gathering methods or procedures are (1^^) 

described 

(19) Data gathering methods or procedures are (19) 

appropriate to the solution of the problem 

(20) Data gathering methods or procedures are (20) 

used correctly 

(21) Validity & reliability of data gathering (21) 

procedures are established 
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E . Data an a ly s is 

(22) Appropriate methods are selected to analyze (22) 

data 

(23) Methods utilized in analyzing the data are (23) 

applied correctly 

(24) Results of the analysis are presented (24) 

clearly 

(25) Tables and figures are effectively used (25) 
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S ummary & Concllisions 

(26) Conclusions are clearly stated 

(27) Conclusi'-ns are substantiatod by the 

evidence presented 

(28) Conclusions are relevant to the problen 

(29) Conclusions are significant 

(30) Generalizations are confined to the popu- 

lation from which the sample was drawn 
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• Form Style 

(31) Report is clearly written ' (31) 

(32) Report is logically organized (32) 
(3li) Tone of the report displays an urbir*sed, (33)!" 

impartial, s.:ientific attitude 
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II. Overall Evaluation (check one) 
(1) Accept as is 



(2) Accept after minor 
revisions 



(3) Accept after major 
X_4) Reject 



revisions 



[II, Specific Shortcomings 



ERLC 



If decision on Part II was to rej ect or to accept after major revisions , 
circle the number of each specific shor!:coming listed under Part I that 
you would cite to substantiate your judgment. 



Description of tlie 
Population of Research Articles 



Tlie criteria used were those of the AERA CcTr.r.ittee cn Evaluation of 
Educational Research^ An article was clasaified as research if it 
contained (1> a statement of th3 problem, (2) the presentation of data 
(not necessarily quantitative data), (3) an analysis of these data, and 
(4) a statement of conclusions. To delimit the journals to be searched 
for educatipnal research studies, the following criteria were used: 
(1) the journal must be indexed in the Education Index , and (2) it must 
be cited 10 or more times in the Review of Educational Research during 
the three year period, 1968-70 • 



Directions on use of the 

Evaluation Sheet ; 

I. Evaluation of Specific Charaeteristics 

Assume the role of a professor of educational research methods 
who wants to develop a series of product scales illustrating five 
quality levels for each of 33 chara -zterirtics of a research article. 
Using the five-point ratin^^ scale beiv^w (representing five quality 
levels of a product scale), rate your research article in terms of 
each of the 33 characteristics on the Evaluation Sheet by plaVing 
^ c heck (n/^) in the appropriate box, on the right. 

Level of Quality 

5 - Excellent 

4 - Good 

3 - Ilediocre 

2 - Poor 

1 - Completely 

Incompetent 

If the characteristic is not appropriate to the study, place an "XV 
in the mar^^^in at the extreme right of that characteristic. 



II. Overall Evaluation 

Assume the role of an editor of a journal that publishes educaticmal 
research: assume that your research article had been submitted to you 
and that you must make one of four choices: (1) accept it as it is 
and publish it, (2) ask for minor revisions and publish it if the revi- 
sions are made, (3) ask for major revisions and publish it if the revi- 
sions are made, or (4) reject it. Under part II of the Evaluation 
Sheet, place a check (s/ ) in the appropriate space to indicate your 
decision. 



De scription 
A nodel of good practice 
A few minor defects 
Hot good, not bad 




A horrible example 



The criteria used were those of the AERA Cotur.ittee cn Evaluation of 
Educational Research • An article was classified as research if it 
contained (1) a statement of the problem, (2) the presentation of data 
(not necessarily quantitative data), (3) an analysis of these data, and 
(4) a statement of conclusions. To delimit the journals to be searched 
for educational research studies, the following criteria were used: 
(1) the journal must be indexed in the Educati on Index, and (2) it must 
be cited 10 or more times in the Review of Educational Research during 
the three year period, 1968-70 9 



Dirfictions on use of the 
Evaluation SVieot 

I, Evaluation of Specific Characteristics 

Assume the role of a professor of educational researcii methods 
who wants to develop a series of product scales il^strating five 
quality levels for each of 33 characteristics of a research article. 
Using the five-poiut rating scale below (representing five quality 
levels o-ff a product scale), rate your research article in terms of 
each of the 33 characteristics on the Evaluation Sheet by placing 
a check (\/^) in the appropriate box on the right. 



Level of Quality 


Description 


5 - 


l^xcellent 


A nodel of good practice 


4 - 


Good 


A few minor defects 


3 - 


IJediocre 


riot good, not bad 


2 - 


Poor 


Some serious defects 


1 - 


Completely 
Incompetent 


A horrible ex; imp la 



appropriate to the study, place an "X" 
right of that characteristic. 



If the characteristic is rot 
in I le margin at the extreme 



II. Overall Evaluation 

Assume the role of an editor of a journal tiiat publishes educational 
. research: assume that your research article had been submitted to you 
and that you must make one of four choices: (1) accept it as it is 
and publish it, (2) ask for minor revisions and publish it if the revi- 
sions are made, (3) ask for major revisions and publish it if the revi- 
sions are made, or (4) reject it. Under part II of the Evaluation 
. Shee"t, place a check (s/ ) in the appropriate space to indicate your 
decision. 



Ill, Specific Shortcomings 

^ (See part III of the Evaluation Sheet) 
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APPEIIDIX D TABLE ID 
The PoDulation and Sanple of Iducatioaal Research Articles PuMislied in 1962 





No. 


of a 


r tides 


Journal In 


populaticn 


Tn GaiTiDle 


Education journals v;hich publish prlmrily research 








articles 




248 


37 


1. Journal of Educational Research 


88 




13 


2. Personnel and Guidance Journal 


60 




9 


3. Science Education 


46 




7 


4. Journal of Experimental Education 


32 




5 


5. California Journal of Educational Research 


22 




3 


Education journals v/hich publish primarily non-researcli 






articles 




281 


44 


1. Bulletin of the National Association of Secondary 








School Principals 


24 




4 


2. Elementary School Journal 


23 




3 


3. Audio-Visual Conmunications Review 


21 




3 


4. Arithmetic Teacher 


2^ 




3 


5. Exceptional Children 


15 




2 


6. Journal of Teacher Education 


15 




2 


7, Elementary English 


13 




2 • 


8» Vocational Guidance Quarterly 


13 




2 


9. Journal of Hegro Education 


12 




2 


in. Junior College Journal 


12 




2 


11. Adult Education 


11 




2 


12. Clearing House 


11 




2 


13. School Reviev; 


10 




2 


14. Education 


9 




1 


15. School Science and Mathematics 


9 




1 


16. High School Journal 


o 




1 


17. NEA Research Bulletin 


8 




1 


18. Volt a Review 


S 




1 


19. Journal of Research in Ilusic Education 


7 




1 


20. Quarterly Journal of Speech 


6 




1 


21. School and Society 


6 




1 


22. Journal of Higher Education 


A 




1 


23. Science Teacher 


4 




1 


24. Educational Record 


3 




1 


25. Phi Delta Kappan 


3 




1 


26. Religious Education 


3 




1 


^^27. tfodern Language Journal 


3 




0 



Related-Profession journals 



98 



1. 


Child Development 


76 


11 


2. 


Journal of Educational Psychology 


50 


8 


3. 


Educational and Psychological Ileasurement 


42 


6 


4. 


Journal of Counseling Psychology 


35 


5 


5. 


Journal of Personality 


30 


5 


6. 


American Journal of Mental Deficiency 


29 


4 


7. 


Journal of Speech and Hearing Disorders 


22 


3 


8. 


Journal of Educational Sociology 


11 


2 


*9. 


Anerican Annals of the Deaf 


3 


0 



44 



All journals included in evaluation of research 



827 



125 
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.*Five journals published only 3 research articles each in 1962. The 15 articles 
from these journals were treated as a single group for purposes of sampling. Only 
3 articles were chosen from this group uith the restriction that a Journal could 
rhe represented by only one article. As a result of this sampling, two journals 
were excluded; 125 articles from 39 journals were included in the evaluation study. 



Table 2]) 

Ratings op. 25 Characteribtics for 125 research Articles (1962) 



Characteristics (listed in order of ii-cai^ ratiiif,) llean SD N 





ieport is logically organized 


3. 


71 


1. 


12 


12 3 


25 


Tone of the report displays an unbiased, impartial 














scientific attitude 


3.69 


1. 


13 


120 


1 


Problem is clearly stated 


3. 


,62 


1. 


17 


122 


3 


Prohlpn is significant 


3. 


,59 


1. 


09 


121 


13 


Data-gathering methods or procedures are described 


3. 


50 


1. 


19 


121 


18 


i'ethods utilized in analyzing the data are applied 














correctly 


3. 


49 


1. 


16 


112 


23 


Report is clearly written 


3. 


46 


1. 


13 


124 


19 


Results of tiifc analysis are presented clearly 


3. 


,44 


1. 


24 


123 


2Q 


Conclusions are clearly stated 


3. 


,44 


1. 


21 


123 


15 


Data-gathering ir.ethods or procedures are utilized 














correctly 


3. 


33 


1. 


17 


115 


G 


Research design is described fully 


3. 


35 


1. 


25 


121 


11 


Population euid sample are described 


3. 


35 


1. 


22 


120 


14 


nata-gathering methods or procedures are appropriate 














to tlie solution of the problem 


3. 


,27 


1. 


09 


119 


22 


Generalizations are confined to the population from 














which the sample '/as drawn' 


3. 


,26 


1. 


22 


111 


2 


Hypotheses are clearly stated 


3. 


24 


1. 


38 


92 


17 


Appropriate methods are selected to analyze the data 


3. 


,24 


1. 


23 


119 


6 


Important terms are defined 


3. 


,16 


1. 


23 


101 


7 


Relationship of the probler.i to previous research is 














made clear 


3. 


13 


1. 


42 


119 


21 


Conclusions are substantiated by the evidence 














presented 


3. 


11 


1.42 


119 


9 


ilesearch design is appropriate to solution of the 














problem 


3. 


03 


1. 


26 


113 


12 


Ilethod of sampling is appropriate 


2 . 


97 


1. 


21 


100 


10 


ilesearch design is free of specific '/eaknesses 


2. 


81 


1. 


13 


117 


5 


Limitations of the study are stated 


2. 


74 


1. 


24 


11. '3 


16 


Validity and reliability of the evidence gathered 














are establislied 


2. 


74 


1. 


3C 


107 


■ 4 


Assumptions are clearly stated 


2. 


73 


1. 


25 


103 



*'nicr>p numbers refer to scales or characteristics rated and can be used to refer 
to data in Table 3£) 



ERIC 



V 



ERIC 









CM 




vO 




rj> 




iH 








0) 








fi 




mo 












o 








0) 




•H 




M 




O 




to 




Q) 




•U 




03 




O 




V4 




D 




O 








Vi 


CO 


o 


0) 


*w 


rH 


w 


,Q 


a 






H 






w 




















a 
















6 








eg 




a 




o 




w 












■ -H 








cd 







\ 



.c 
o 

M 

cd 

OJ CO 

H 0) 

1 rH 

o ^ 
Ci f J 
u 

<3 

w 

'"1 r-1 <r 

o 

'r-> 

















o 












cd 


0) 


§ 












■U 




a 


CO 


M 




u 




4J 










03 


<: 




»H 






CO 






n 


CO 










=3 






O 











o u 

O <\ 



ir 

















ma 


CO 




:3 


C) 


1 


o 


H 




•n 


U 


<U 




•H 




n 


-U 


cd 


o 






•H 




Q) 


cn 






CO 






<u 


<r 




CM 






o 






M 











cn 
o 

I H 

cd to 



cNjcj<fCvirH'^cn-<rr^c^sou^cMcxDCO>*<r<7>r^o.^coo^in\o 



I 



i-H CN I i-H ^ 



fH CM tH CVI rH rH 



<rcsj<r^fncNcn<r«^<^>3'c^^>^<fro<fc^<t^<j'<J:<t<3'>^ 

iHsrOOrHCslCN'HrHOfHcvJfHO'HrncNrHrH'NCNCNOOf-l 



rH rH rH 



rHT-HrHr-HrHr-HfHrH 



vo o <t '^o v£> o c^ ^^o CO in Lo cvj CO T-H in r> cNi 00 c.^ c^ o o 

CO o> cNj <r cjN m m CO a> o'^ CO CO inxr^ co c:o <r o> cm cn 

cncsiC^CMCJCMCMCNJCMCMCMcMCM CMCMCMCNCMCMCMCMCMCOCnm 



v^,-^^;^<a•^Ovo^c3^cMOcocMo»^oc^<3^c^c^^^cococ^^^ 

CMtnrHc>JHCMMrHOO»HCsIrHc^O>^'^iHCOiHOC^rHrHOO> 
^,-^r^r^^HTHr^r^r^r^rHrHr^ rHiH rHrHrHrHrHiH 



CM CO cn tn CO o o> ^ c> in r^. G> CM V.O CO CO ,H o o\ <^ 
^^coincor^voc^r^vo^Ar^rnrHrHr^rHcN^-cMrrkCMrHinvo 

C^CO(^CMCMCMCM^C^CMf^CMCnc^rOCMf*^C^COOcMf^cnrOfn 



:i <c x -x •« •?< * ^ -vi -y. ^ -i^ -ii -ic -jc 

H< 'Ji -li -A -A -i< -!C -iC -K -JC Hi -^A 'V: -ic -ic >C -'.^ 

mcN'^"^f*>*^»x>Ln'HcMCMfnooovocoo^o iHcocoqoco 

p^sjooco,H^OjOmc^i»Hv^'^o^«n «Hcr\HO ^<i"*<i--<rr^ 

cv4fH<tro^cn\o<3'<J"i*'^csi»-H<f<j'<rc\iin<r>rrin mcMc^cnco 

C0-:tc0C0vDCMtn00rHinC0r000v^C0C0vX5OC^<Tv VOU^OCT^f^ 



CO<rC'>OOinrMrHC7Nf^<fvOvOvOrHOr^CMCMCOvD CM<tvOlACM 
rH^i'OrHr-lCMCMrHrHO'HcvlrHOrHCOcMO H rH CO CM O O iH 



rH rH iH rH iH rH 



rH rH rH rH rH 1-t rH 



rH<trHO'-H<fO^LncMCX)in<fO">rHC^COrHrHvD fOr^rH MOCM 
vi-OC^<t^COvDOvO<3''-*COrHCNO<-OOrHrHO vOOCM<r-J' 



rOCOrOcMCMCMCM^CMCM^CMCncMCOCMCM<^ CO^O cm CO CO CO CO 

N 

>d'COC')OCMCT\N^COCMfMCMLnc^C*^CMCT\COCM<rN3' COvO*^>d'<^ 

<tco<t<i-<rco^<-^^<-co-vr>d-^fo<i'<r'<r<3' •<3-c'-»^<r<- 

cMrHCO*srr^tnincoo>0'J^<tcncovjococMr^cvio cotnocr^ 

OCMCOfH.HOCMrHrHrHcMOCyvOaNCMOOrH<7\ •HOf'.O'n 

• • • • • • • • • <T> 



O^C^QC^f^CMC^r^-Or^c^-^r^cMOOOOcMCMiH ln^^rHvOO^ 
Ouoo<'^<^v£)Oc^vCtnvoc^JrHr*«xOl-^a^r^ orH c»n vo on rH rH 

<tco<fcoroco<-coco cocOco^co<J-coro<r >d-<r co co co <f <»• 



rH 

rHcM<^*<l"'J^vOt^COcrs O'HCM^^srinv0^^0Cc7^O»H<MC0^m 
,-4 f-H rH rH rH rH rH rH r-l rH 'M Cvl rJ CM CM C^I 



> > 



in tH 
o o 

• • 

<d cd 

a o 
•H ^ 
mh mh 

•H ^ 



CO 

o 

tH 
'J 

•H 
■u 
U 

O 

Qi 
CJ 

o 



o 

M 

H CO 
CO (1) 

(U 

o I 



(0 



C2 

O W O 

4J td r} 

CO d c) 

a u 05 

t:) o M 



a 

O CO 

•H 1-1 

4J CO 

o u 

:3 :3 



I o ui 

-r! -H rH 

O 0^ td 

4-* 03 0 

CO (U M 

o o 



:3 



CO 



00 



in 
eg 



CO 



u 









































m 


LO 


























rH 




M 




m 


LO 




CM 




in 


CO 






rH 




■K 


to 




• 


tO 


nd 














N 




















0) 










, 1 1 








at 


ed 
















<u 










\o 








u 


•H 














O 


•u 


M 


















to 
















<u 








11 








C) 




s— ' 














■c 


CO 










( 

1 






CO 




CO 












o 


QJ 








o> 






vH 


CO 










to 

rH 


in 




les t 


uodi 



to 

0) 



to 























CM 


tH 






UO 


O 
















V 








P 


* ^ 


CO 








CO 




































lO 
















<r 








CM 


SsJ 


^1 






rH 


1-H 


c:0 


11 


<1' 




t-H 










CM 








X 


CO 


CO 






rH 





















a\ 


rH 


O 


r-H 


<r 


<^ 








<t 


iH 


o 


CM 


m 


in 





u 


a 




0) 




CO 


CO 


ti 


(U 


IS 


ec 






CO 


o 


> 




o 


O 


M 


o 




Re 








< 















tH 






o 












V 






a 












CO 






CO 












m 


vO 




n 


in 












rH 







CO 
C^I 



CO 







CO 




CO 


a 


(0 


CO 


to 






0) 


c 


•H 




•H 








CO 




to 




o 


)^ 










(1) 






a 












o 










o 


a 






«4H 


CO 








Q 






CO 





D CO 

cr <u 
u 

u o 

•H 4-1 

o 

^ 'a 



I 

0) 



P 

CO 



a 


^0 






CO 




o 


CO 






CO 


CO 






tH 


'H 






(U 












iH 




O 












CO 








CO 




> 






•H 






4^ 


CM 




CO 






0 






M 


CO 




<U 






4~» 






rH 


o 




CO 








(0 




0) 


CJ 




a 


a 




u 


•H 












to 


















rH 


4J 


4^ 


to 


O 




U 


(U 




•H 




CO 




(U 


CO 






c 






•H 






iJ 


:3 




CO 


c 


0 






CO 








r: 


u 


«t 


to 


•H 




a; 


►> 








CO 






•H 


CO 


a> 


> 




rH 


(1) 




to 




4-> 


o 






CO 




a< 






e 


CO 




o 


c 




u 


-H 








<U 






u 


o 


2 


u 






< 




4^ 




CO 


c 




•H 


•H 


CO 


CO 


O 




to 




e 




1 


o 




0) 


4J 


to 


0) 






}^ 


a) 


CO 




<: 


4J 





(U 

•H 

0 
o 
u 

0) 



CD 
(1) 
•H 
VI 
O 
CO 
(U 
4J 

to 
u 

(U 
CO 

> 

M 

T) 

0 
CO 



o 

u 

CO 



o 
CO 

0) 

to 

4J 

a 

(U 

c» 
o 
to 



CO 

a 

to 



<u 

CO 
0 

to 
o 

0) 



